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ThU  paper  jr.dloote*  how  a  isahniaye  which  eluatsrs  criteria  in  terms  si  the 
homogeneity  of  their  prediction  equations  ass  be  used  to  identify  and  describe  the 
ratliii  pelieise  within  a  group  or  board  sf  judges,  The  technique  measures  the  eon- 
eletency  of  rating*  obtained  from  individual  judge*,  and  indicate*  the  amount  md 
nature  of  agreement  between  judge*  or  group*  of  judge*,  A  practical  method  U  *ug« 
gsstsd  for  helping  a  beard  or  semmlttes  to  reaeh  a  consensus  concerning  how  relevant 
faotore  ehall  be  weighted  In  future  eituatlone  no  a*  to  carry  out  it*  final  joint  policy, 
Examples  are  given  to  *how  how  the  technique  can  he  used  to  determine  the  oocipoel- 
tion  of  a  service  eohool  leleotlon  eompoalte,  to  d*t*rmine  factor  weight*  for  a  Job 
evaluation  plan,  and  to  determine  how  relevant  factor*  ihould  b*  combined  into  a  Job- 
oriented  criterion  composite, 
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JAN!  A  TKClfMJQUE  FOR  AHAI-.Y"INf5  GROUP 


Jf  inOMRNT 1 


!<f\W  t  Judgment  ANtilywIe)  1ft  •'  '  't'.jilm  hut  powariul  technique  far  Identifying  arid  describing 
th#  fating  poltcioB  that  ftiilst  within  a  board  or  eommittes  aS  Judges.  It  ma  be  used  to  help  the 
members  of  suoh  boards  or  committees  reach  a  consensus  and  will  eNpreas  their  final  joint  policy 
In  a  pro  else  manner. 

JAN  U  a  npscial  application  of  a  technique  developed  by  Bottenberg  &  Christa!  (39S.1I  which 
groups  criteria  in  term*  of  the  homogeneity  of  (heir  prediction  equation*. 9  The  criterion- grouping 
technique  1*  a  mpseisi  application  of  a  still  more  genera!  hierarchical  grouping  model  developed  by 
Word  (1981}.  The  reader  i«  referred  to  these  paper*  for  computational  expressions. 

Communication  of  the  JAN  concept  eon  he  simplified  by  describing  how  it  might  be  applied  in 
a  hypothetical  situation-  Suppers*  ons  were  fused  with  the  problem  of  selecting  the  tat  clans  of 
300  students  for  a  newly  established  service  school.  Scores  arc  available  oil  IS  selection  variables 
for  eaoh  of  SOHO  applicants.  A  special  board  of  10  officers  has  been  assembled  and  charged  with 
she  responsibility  for  determining  how  these  variables  shall  be  weighted  together  to  compute  the  final 
selection  eompoeite.  Since  this  in  a  new  school,  criterion  dote  ore  unavailable.  However,  eaoh 
board  member  l*  familiar  with  the  curriculum  to  be  offered  and  with  the  general  characteristic*  c-i  the 
selection  variables. 

In  order  to  set  the  stage  for  u  discussion  of  JAN,  e  portion  of  the  proceedings  will  be  fabricated. 
The  letters  MA,  MB,  and  so  forth,  will  be  used  io  refer  to  the  individual  hoard  member*, 

The  board  hat  beep  sailed  te  order.  The  chairman  has  recognised  MC. 

MCt  MB  end  1  would  like  to  prsjeee  a  ooi  of  weight o  lor  the  hoard's  considers;! Jon.  If  these 
weights  are  ueed  as  a  beginning  point  isr  our  dissuasion,  perhaps  w*  oan  proessd  more 
rapidly, 

Copies  of  the  weight*  are  passed  cut  to  all  hoard  mambara,  Members  study  the  proposed  weight*. 
MAi  What  concern*  me  is  the  low  weight  you  have  assigned  the  High  School  Activities  Index. 

Afier  all,  it'e  the  only  vatlabla  in  the  selostlsr,  battery  that  measures  leadership. 

MCi  But  we  gave  it  a  weight  of  three,  and  that's  pretty  high.  It's  almost  as  much  weight  a*  we 
give  to  all  of  the  verbal  teste  combined. 

MAi  Ye*.  But  th*  highest  Activities  hide*  score  is  only  300,  while  the  lowest  scores  on  the 
verbal  tests  art  unually  higher  than  300.  1  think  we  should  weight  the  Activities  Index  by 
at  least  20. 

COMMENT)  Notice  that  this  ie  a  very  powerful  argument,  evun  though  It  1*  based  en  (also  promises. 
Th#  board  at  this  point  does  not  rsoognlst  that  the  affect  of  an  applied  weight  Is  Influenced  by  the 
standard  deviation  of  the  scares  to  be  weighted  and  is  not  n^aoexarily  r-  Filed  to  the  mean  or  else 
of  the  scores  to  be  weighted. 

The  hoard  compromises  on  a  weight  ol  15  points  lor  the  Activities  Index.  The  conference  con¬ 
tinues,  Various  members  argue  lor  Inareasing  or  decreasing  the  weights  lar  certain  variables. 
Compromises  are  made  until,  finally,  a  tentative  set  ol  weights  In  agreed  upon,  As  a  check, 
th nae  weights  are  applied  to  tho  scares  fas  a  sample  ol  the  applicants.  Certain  peculiarities 
are  observed,  which  are  taken  up  during  the  next  day's  mooting, 


^  nsprnrfuowfi  with  iinme  minor  odituriai  a  linn  40  m  from  n  papt»r  nrwRnnied  fit  Him  minimi  mastlrin 
of  ih«  Smith wantarn  Pnyfihfiloqlnnl  Annoniniton,  FoM  Worth,  Tgh«ii,  <1  Aoril  1(162, 

Dr.  .Tr*3  II.  Wise.!,  Jr.  •«  t?r*»dl:£ri  with  rmijqButJmi  mm  «f  a  lount-mjunroft  w&{slitn<i  ToqroMiilrin 
Immiila  lo  capture  the  pet  liny  nl  a  iilmjln  ratwr  (qln«  llolimfiri,  HSR),  Th«  prnnnnt  pftpnr  IwllmUnn 
how  thn  noM®nb«ru-^hrlwi«i»  nrllorlnn  ffmupfnq  ti*nhnl(|iio  oan  ho  to  «  irnrUn  0/  Much  mjnnUum; 

hi  nniai-  t«»  ««n*»!y(f.s3  Urn  Tiiiimj  linlinvioF  i»f  n  qrnup, 
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weighting  nyi'itc'iu.  Yon  v;l!l  n)m«vvr<  Hint  severed  nl  (In' 


applicants  with  thn  hlgbenl  i;oinpnslt«  scores  made  lens  than  401!  on  mathematical  aptitude. 
I  don't  think  wet  nhnuld  lot  in  anyone  who  sinornn  below  451)  on  thn  Mflthoniatlnn  Aptttu  lo 
Tont.  They  will  never  ptimi  our  uiqliwering  curriculum. 

I  think  thn  problem  l»  Hint  wr  are  weighting  the  AeHvtlian  Index  loo  much.  Yon  will  notion 
that  all  of  those  cases  with  low  quantitative  ssores  how  high  Activities  Indexes, 

Well,  i  don't  think  w»  should  lower  tho  wslqht  (or  thn  Activities  Index,  It'n  tins  only  maim- 
ur«  oi  leadership  we  have,  Let'  ,  establish  fluted*  on  the  verbal  an  it  quantitative  tests; 
then  we  will  rntissa  to  accept  any  candidate  who  scores  below  our  established  cutoffs. 


Thu  hoard  aomprain'ises  by  lowering  the  weight  on  the  Activities  Index  to  l?.,  and  bv  antahll thing 
outalts  al  4S0  an  the  verba!  and  quantitative  hut  a.  A’ew  composite*  are  computed  hr  the  sample 
acmes,  and  the  heard  in  ^convened. 


MEt  I  don't  like  this  multiple-cutoff  technique.  I  found  several  applicants  who  were  high  on 
newly  all  variables,  but  who  ware  dtsquallfisd  because  of  a  single  verbal  or  mathematical 
score  of  440  or  445,  We  should  make  an  e.ospuoii  and  iei  these  oases  In.  Alter  all,  how 
much  difference  is  there  between  a  score  of  440  and  one  of  450? 

MFi  I  still  think  we  are  weighting  the  Activities  Index  too  much.  For  some  reason  It  seems  that 
an  applicant  with  high  scores  on  alt  variables  except  th*  Activities  Index  comes  out  with 
a  low  composite  scorn, 


Alter  a  prolonged,  discussion,  the  chairman  decides  that  the  weight  lor  the  Activities  Index  will  be 
lowered  to  8.  and  that  the  avtoll  teams  will  be  lowered  to  400, 

This  conversation  hoc  bean  carried  (a  enough  to  make  clear  oertaln  points.  First,  the  board 
Wee  unable  <o  predict  accurately  the  effect  of  their  proposed  weights,  Very  early  In  th#  conference, 
a  tentative  set  of  weight*  was  agreed  upon,  However,  no  ons  was  satisfied  with  th*  results  when 
these  weights  ware  applied  to  the  applicant  group,  Recommendation*  from  this  point  on  were  based 
primarily  upon  the  Individual  board  members'  satisfaction  or  dissatisfaction  with  ths  rank  position 
of  applicants,  If  thsy  full  that  particular  applicants  ware  ranked  too  high  or  too  low,  reoommtmda 
(fans  were  mad?  for  changing  weights  in  a  manner  predicted  to  rectify  the  problem.  Often  as  not, 
the  proposed  sbangesi  either  failed  to  resolve  the  problem,  or  areated  now  problems, 

When  one  carefully  analyses  tho  board's  cations,  he  eomos  to  realise  that  th*  board  should 
not  have  been  asked  to  propose  weights  ir.  ths  first  pi  do*,  Its  main  ucnoern  wan  that,  whatever  the 
weights,  the  final  selection  composite  would  rank  applicants  In  a  manner  which  Is  acceptable  to 
the  board  members.  Rather  than  argue  over  the  relative  weights  to  hs  applied  to  the  selection 
variables,  the  board  should  have  restricted  Its  attention  to  the  relative  acceptability  of  applicants 
with  given  profiles  of  scores  on  the  selection  tests,  The  Impact  of  applied  weights  on  the  final 
rank-ordering  of  appIlaanU  1*  a  hiqhly  complex  matter  which  1*  Influenced  by  several  thlngn,  Includ 
Ing  ike  standard  deviations  and  rmeraorreiaUons  among  the  selection  variables,  A  sophisticated 
measurement  psychologist  cannot  solve  for  a  15-predlctor  regression  equation  In  bin  head.  Yet  thin 
1h,  In  offeat,  what  was  required  of  the  hoard  members  when  they  were  asked  to  determine  tho  weights 
to  ha  applied  to  tho  selection  tests. 

Nor  should  the  board  memhnrs  have  attempted  to  establish  cutoff  scores,  It  may  he  true  that 
appllcantn  below  qlven  aaora  levcln  on  certain  variables  lend  to  become  unacceptable,  requrdl.-mfi 
of  their  scores  on  other  varlahles,  However,  there  In  no  clear  basis  for  deciding  where  these  cutoff 
points  nhuuld  ho  established,  As  ML  put  It,  "After  all,  how  much  difference  Is  there  between  n 
scorn  gf  440  and  one  of  450?"  What  the  hoard  really  wants  Is  for  these  variables  to  he  weighted  no 
that  applicants  obtaining  scorns  below  certain  points  will  have  u  rnpldly  deoreaning  probability  of 
receiving  a  toil’d  composite  scorn  high  enough  to  bo  rmooptnd  Into  thn  school.  Tho  dofilved  mini!! 
could  ho  obtained  hy  Introducing  appropriately  v/nigntf'd  squared  tom  nrnrnri  Into  tho  (redaction  com 
nostro,  but  It  tv  hiqhly  Improbable  that  the  hoard  would  enmo  up  with  Much  n  suqgnntlon,  Again,  nun 


wrlvnM  iri  the  Rom;]u«mn  that  thn  hoard  nhould  roach  agreement  on  list-  ntipronrlfitr*  rent  p>?e!<!o>> 
sf  '"fif-iJemin,  end  ionvo  rim  (inlfirtl.-m  vmiqhtp  to  tin  ilntarmineH  by  analytical  liiothoilri, 

rinw  that  the  proper  groundwork  has  henn  laid,  th»  timer  haw  corns  to  deocribs  haw  the  JAN 
technique  can  help  such  a  board  reach  closure.  The  first  Rt«p  !s  1°  prepare  a  card  for  oach  of 
approximately  ISO  men  selected  at  random  iroin  the  total  applicant  group. 3  The  card  lor  each  man 
llnte  li.ln  15  selection  test  booms  and  hU  identification  mimber-nothisg  mors,  irrevelant  informa¬ 
tion  which  might  bln*  the  rater,  such  as  the  applicant's  nos-, a,  religion,  or  home  town,  is  omitted, 

Next,  every  member  o{  the  board  is  given  a  set  of  these  cards,  Board  members  are  asked  to 
study  the  test  score*  for  each  af  the  150  applicant*,  and  than  rank  them  from  1  to  ISO.  That  Is,  a 
board  member  place*  the  card  on  top  which,  in  hie  Judgment,  describe*  the  applicant  most  deserving 
to  b*  accepted-end  no  on.  Th*  card  he  place*  on  th*  bottom  1*  assigned  a  rank  of  150,  and  deNcrlben 
the  applicant  who,  in  hi*  judgment,  I*  least  qualified. 4 

Now  let'*  consider  thl*  Information  for  an*  of  th*  rater*.  After  oarefully  studying  th*  eeieo- 
tisn  sears  information,  f.iF  has  renk-ordernd  th*  «cmpi*  ai  ISO  applicant*.  MF's  rating  policy  can  be 
captured  by  u  sin  if  the  aslsctlan  test  vatlaklefi  at  predicts*  end  hit  ranking  o  I  th*  i  50  applicants  as 
a  mlttrlan  variable,  This  simply  require*  computation  of  a  15-predlctw*  regr***lon  equation,  The 
magnitude  of  th*  obtained  squared  multiple  ijorrelaHos  coefficient  (fi^)  indicate*  how  well  MF's 
poiioy  aan  be  r»pi  touted  (in  this  eampl*}  using  K  l  «f»«S1in'a*tfuvN wS'rrSi^fiiGu  fcJOmpdMitft, 

Observe  that  MF  had  to  render  hie  judgment!  solely  on  the  basle  of  selection  test  soars  Infor¬ 
mation.  Nothing  *1**  was  provided  to  him.  Under  these  condition*,  h*  could  not  have  been  blaeed 
by  Information  in  the  applicants'  folder*  which  the  bead  ha*  Judged  to  be  lrrevelant.  Perhaps  of 
more  importance,  however,  I*  the  lout  thet  on*  should  h»  able  to  do  a  good  Job  sf  accounting  for 
MF's  evaluation*  ufllng  th*  Information  which  he  studied.  In  fact,  there  are  only  two  reason*  why 
an*  should  not  b*  able  to  predict  MF's  judgment*  perfectly  using  thee*  data, 

First,  it  iftoy  be  that  MF  coneidered  eertaln  Interaction*  among  th*  predictor*,  or  that  he  per, 
oeived  the  relationship  between  asrialn  of  the  factors  and  applicant  acceptability  an  being  nonlinear. 
Under  these  figsdlu*  •«,  it  would  not  be  poeelble  to  account  for  MF's  assigned  evaluation  scored 
unities  the  appropriate  product  or  power  terms  were  Introduced  into  the  equation.  However,  it  le  a 
siirple  matter  to  Introduce  any  euoh  term*  which  are  hypothesised.  For  example,  If  it  wet*  hypothe- 
nixed  that  MF  consider*  appllaant*  who  seared  below  certain  value*  on  th*  verbal  and  quantitative 
tents  fis  Luting  unacceptable,  regardless  of  how  they  icored  on  other  test*,  then  squared  terms  are 
introduced  Into  the  equation.  If  Introduction  af  iquared  term*  cause*  a  significant  inartase  In  pro 
dlollva  efficiency,  then  th#  formula  for  expreeolng  MF'*  policy  ha*  been  Improved,  Notice  that  thl* 

1*  a  much  better  approach  than  simply  arbitrarily  establishing  cutoff  score*,1 

A  second  ami  perhaps  mare  likely  explanation  ior  an  Inability  to  account  for  MF's  Judgments! 
la  that  he  was  Inconsistent  in  bis  rating  behavior.  Such  Intrsrsisr  Instability  will  lead  to  u  reduced 
value  of  the  obtained  Hs,  To  the  extent  that  MF  1*  consistent  in  hi*  rating  behavior,  it  should  be 
possible  to  produce  a  formula  which  will  capture  his  policy. 

Bui  MF  1*  only  one  of  the  10  members  serving  on  our  hoard.  The  goal  Is  to  obtain  a  hoard 
policy  on  «eleDt!cn™not  on®  man's  opinion,  in  order  to  guide  the  board  to  closure,  tho  ilrnt  atop 
In  la  compute  10  separate  rcgrasnlon  equations.  Each  of  these  10  equations  expresses  the  policy  of 
one  of  the  10  judges-and  each  equation  could  be  used  to  rank  tho  entire  5000  applicants  in  a  manner 
which  would  please  at  least  one  of  the  10  board  mnmborn. 


Tha  mi  in  h  a  t  n!  nniiHH  In  ho  Httleuilwd  <lt*jt«nr]«  on  th”  number  M  nEjlftfiUnn  vnriuhlmi  hfllmj 
mved.  A  gnnl  In  to  cv«*d  "ovm-fMttne"  whim  thnun  varJahlnii  nr«  w»lifht»»d  into  n  I /Mint  p.rjmirmi 

rP  tti]Ufillftri . 

4  It  would  l»n  pnrmlHnihhi  5n  huvo  thmm  nnnon  ;alml  liiutnad  nf  rmiK 

*’  Thw  ooi»ibin«ttnn  ot  n  j»rop«?rly  wolqhlnd  f'/morod  one'  J  i  j  i  i  *  *if -!  fjv  Is  tin;  !ho  »,)>lllty  In  Hi 

t ;  rouvennlaii  liuw  t»»  «my  hH  *>»  pninltt  which  dlHplcyn  «i  ningln  hf»»n  or  nhanqo  in  dlrnutlnn- 


beginning  with  thene  aquations,  the  eritwkm-gKmptnq  tMthnlqiw  Is  »mninwri  to  dcfirtc  the 
iwoph  of  agreement  cr.d  disagreement  among  Ho  i.r.wtj  members,  A a  t|iS  f lrtaS  step,  C  «?>«•■  VHlif® 
ol  fl*  le  computed  which  Indicate*  the  overall  predictive  ofilol«iwy  obtained  when  «  wparaie  Uant- 
sifttctjss-welghiad  mgr? onion  equation  is  used  iof  such  rater.  During  the  socwtd  step,  sash  equa¬ 
tion  Is  compered  with  every  other  aquation,  In  this  way,  ths  two  raises  ©g  located  who  are  in 
closest  agreement  oonusrnlng  how  ths  selection  vsarlahUa  »hou!d  be  weighted  to  determine  ths  rela¬ 
tive  acceptability  of  applicants.  Thee#  two  raters  have  the  meet  hnmeg»nnn»s  r«nr«9B!9n  equations. 
The  computer  then  determines  the  single  equation  that  beet  represents  the  joint  policy  of  these  two 
relate,  It  also  indicates  the  overall  loss  in  predictive  efficiency  that  results  when  the  original  N 
equations  are  replaced  by  ths  host  set  ol  N-l  equations,  This  is  accomplished  by  comparing  the 
magnitude  of  the  overall  ft*  in  the  two  Instances,1 

In  subsequent  operations,  the  program  systematically  reduces  the  number  of  prediction  equa¬ 
tion*  (or  rater  clusters)  by  ons  at  each  step  so  as  to  minimise  the  loss  In  overall  predictive  effi¬ 
ciency.  Ip  each  instance,  ths  computer  prints  cut  an  overall  ft*  at  well  as  the  single  equation  far 
each  cluster  which  best  represents  the  joint  policy  of  all  raters  In  that  cluster.  At  the  last  step,  a 
single  prediction  equation  is  generated  so  as  to  express  the  joint  policy  of  all  refers  with  as  little 
error  as  possible, 

Examination  of  the  loss  In  overall  predictive  efficiency  at  various  stages  of  the  grouping  proass 
will  help  to  identify  the  number  of  ra'lng  policies  existing  snsng  the  board  members.  For  example,  ii 
very  little  overall  predictive  efficiency  Is  lost  in  going  from  ten  to  two  equations,  but  a  considerable 
amount  at  efficiency  is  lest  in  going  from  two  to  one  equation,  there  is  strong  evidence  that  two  dif¬ 
ferent  rating  policies  are  operating.  That  le  to  say,  some  of  the  board  members  are  using  one 
weighting  system  and  the  rest  a  different  weighting  system.  Of  course  ths  hope  is  that  a  single 
equation  can  k*  found  which  predicts  the  criterion  values  assigned  by  all  raters.  In  this  ease  the 
beard  could  be  dismissed  without  further  ®le,  since  their  eingle  agreed-upon  policy  would  have 
been  determined.  However,  should  mare  than  one  rating  policy  be  found  to  sxlet,  tho  board  is  not 
asked  to  arbitrate  the  differences  In  weights.  Rather,  the  several  equation*  are  used  to  rank  the 
applicant  sample,  and  the  board  is  asked  to  arbitrate  the  differences  in  the  rank  position  of  canes 
resulting  from  application  of  these  several  equations.  Ones  a  single  agreed-upon  rank-ordering  of 
the  sample  has  been  obtained,  It  is  a  simple  metier  to  determine  the  appropriate  set  of  weights  for 
the  final  selection  composite, 

Ths  JAN  technique  enables  ths  investigator  to  make  s  complete  analysis  of  lnt*fcstes  agree¬ 
ment.  As  a  matter  cl  fact,  results  of  the  first  step  alone  indicate  the  amount  ol  agreement  between 
each  rater  and  evsry  other  rater, 

We  have  seen  that  a  situation  may  exist  vhsro  the  levs!  of  interrater  agreement  among  the 
entire  sample  of  rotors  in  lew,  but  where  the  raters  oan  be  divided  into  two  or  more  groups  within 
each  of  which  there  is  high  agreement,  Conventional  analysis  techniques  for  determining  lnterrotsr 
agreement  would  not  detect  this  situation,  The  JAN  approach  will  not  only  Identify  ths  groups,  but 
will  at  the  same  time  provide  equations  expressing  the  different  rating  policing. 

So  far,  only  one  example  has  been  given  illustrating  how  JAN  might  be  used.  But  the  Air  Force 
Is  constantly  calling  beards  together  to  determine  how  variables  should  b#  weighted  for  making  deal- 
■fens!  Who  should  be  promoted?  What  specialties  should  receive  proficiency  pay?  Who  should  he 
given  n  regular  commission?  What  arc  ths  grade  requirements  for  jobs?  Which  officera  should  be 


1  Wilt, I  the  ti.Oiiully  it i>e n  during  this  Unit  step  (H  ta  evaluate  enoh  ol  the 

nltnrnnto  ways  nf  redlining  lli«  immhcvr  of  equnlUir.H  from  A'  lo  N'l,  Any  nno  of  liuitiu  wfiyn  InvfWven 
p  re  mutiny  *!*«  *?rlt*rinn  ohnrtn*i<finnH  for  «  pwtinulor  pfilF  f,i  mumhorn  ivflh  n  ninqln  nqnqllnn, 

•vhils  ysininw**!  riupurala  lauri  miuin«Mwt*lqhlftd  ntjunUnr.H  for  #»flnh  nf  the  rsmatninq  /V  C  hnarr! 
mtvnihprit.  The  end  rnmiH  In  liiuutlnnl  tn  th<i(  which  wntiKd  he  rrhlriitinH  by  the  prnnnduro  outlined 
h*  »htf  tiitdnr  *!ir  rnnrHIInnjt  net  forth  In  thin  pop  Hr. 


qlvsii  responsibility  pay V  The  nnlutinn  to  nil  such  prohlflimt  involved  either  the  implicit  or  explicit 
weighting  together  of  factors  judged  to  bo  rnlevaut  {or  noivluing  thn  ogreeit-upm!  goals.  If  thsno 
factors  aw  made  explicit,  then  JAN  eon  Ha  uead  to  h dp  such  boott'o  «oah  a  consensus.  Moreover, 
the  JAN  approaah  romiltn  In  a  precise  statement  of  the  factors  and  welghte  to  b a  used  in  carrying 
out  the  Hoard's  recommendations, 

Very  often  n  group's  policy  aon  be  aaptured  by  having  It  rate  a  series  of  simulated  situations, 
For  example,  Tbs  Ohio  Stats  University  nssesuh  Foundation  i*  conducting  a  job-oriented  criterion- 
devslopment  contract  In  which  soar**  on  hypothesised  crltt?lon>relevant  factors  ore  simply  naorlbsd 
to  e  sample  of  several  hundred  pseudo-incumbents  presumably  working  In  a  particular  Air  Ferae 
specialty,7  Supervisors  in  this  specialty  will  be  asked  to  study  the  resulting  score  profiles  and 
make  a  global  evaluation  of  the  worth  or  effectiveness  of  each  simulated  lncumbsnt.  Thsn  the  JAN 
technique  will  be  applied  to  determine  whleh  factors  should  be  weighted  into  an  eaeeptgbie  crite¬ 
rion  compsslts  tor  use  In  the  [specialty.  This  approach  to  criterion  development  has  the  advantage 
of  not  requiring  expensive  field  testing.  !i  also  makes  possible  the  evaluation  of  factors  which  as 
the  present  time  cannot  be  measured  easily  in  the  field.  Far  purposes  of  the  study,  it  is  only  nsoes- 
■ary  that  each  racier  be  definable  in  terms  that  present  unambiguous  concepts  to  the  supervisors 
who  must  make  the  ratings. 

Job  evaluation  is  another  area  to  whish  the  JAN  technique  has  been  applied  (Madden.  IASI). 
Work  requirement  factors  wero  developed  as  likely  candidates  for  Inclusion  in  a  revised  Air  Force 
offloer  Job-evaluation  system,  The  {gator*  cover  such  requirements  as  formal  sduoatlon,  special 
training  end  work  eepsn****.  tx** io  jtmni  responsibilities,  and  working  condition*,  A  sample  of 
Air  Forae  Jobe  was  rated  sm  these  factors,  and  a  sesrs  profile  for  sash  job  was  gensratsd  by  com¬ 
puting  its  mean  rating  on  eaah  factor,  Senior  officers  at  the  Air  Faroe  Commend  and  Staff  College 
were  aeked  to  study  thess  profile*  and  rate  each  of  the  corresponding  jobs  on  the  basis  of  d**«rvnd 
aompsnsatlon.  In  eubssquerii  analyses,  the  JAN  teahnique  was  able  In  most  instances  to  capturs 
between  30  and  Li  percent  of  the  variance  In  the  ratings  mads  by  these  individual  of  flours.  Further¬ 
more,  a  great  deal  of  similarity  was  found  In  ths  raters'  pollutes  concerning  which  factors  should 
determine  the  pay  lave!  for  Air  Foroe  jobs. 

The  applications  of  IAN  suggested  in  this  paper  have  related  to  Air  Foroe  problems,  but  It 
should  be  apparent  that  iU  technique  Is  generally  applicable  to  any  situation  whers  a  committte 
or  board  wish**  to  sxprsss  It*  policy  eoncurning  how  relevant  factor*  shall  he  combined  for  making 
subsequent  decisions. 

The  teahnique  also  aan  be  used  to  analyse  and  describe  the  similarities  and  differences  among 
ratings  ealleotod  from  Individual*  who  are  separated  by  time  or  space,  anti  who  in  no  way  can  bo 
considered,  a*  being  members  of  a  board  or  committee.  Perhaps  it  oould  be  used  to  analyse  cus¬ 
tomers'  preferences  for  combinations  of  commodity  characteristics  or  to  analyse  voters'  preferences 
for  certain  characteristics  in  rsal  or  eimulatsd  political  candidates, 

Finally,  It  should  he  recognised  that  application  of  the  JAN  technique  to  large  problems  le 
not  feasible  without  use  of  an  elsetronto  data  processing  machine,  However,  most  Investigators 
now  have  aoeess  to  computers,  and  studies  of  the  typo  discussed  are  easily  accomplished,  If  a 
board  of  10  judges  each  made  150  evaluations  based  on  15  factors.  It  is  estimated  that  a  complete 
analysis  of  the  data  using  the  JAN  teahnique  would  take  less  than  five  minutes  using  a  modem 
computer  comparable?  to  the  IBM  7000.® 

7  As  nno  Invuutlqalnr  nloteit  In  |»h!i  "Ws'rv  tlrod  nl  bsliic,  unable  Ia  predial  thn  behavior 
nf  rncchlivn  pwnpln,  mo  wo  rlanlrlnri  to  nrnnlpj  nnir.e*  o!  n»r  own. ft 

®  Qualified  rsqiienlnrs  can  oiiinln  a  eel  of  J’orlran  neraputlnq  utatsivsiilH  Inr  Urn  ailtorlen- 
f|pnttnltt<7  tfirthnlqilf*  hy  wrttlno  tn  tho  fiSVOlh  Ppmonnnl  flnnofnrh  I.nlwirfitciry'.  Thin  '/  nrtrnn  pfonrnm 
•.Vffyi  wrlflAit  hy  Mr.  Puninl  FUqnny  nruUf  thn  qonnrnl  rtirnwtinn  nf  Hr.  W  Wnrii,  »r. 
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